Generally, augmented and virtual reality technologies allow users to experience content in three-dimensional (“3D”) using some kind of display (e.g., in a head-mounted or hand-held device) capable of presenting virtual 3D environments from different perspectives. For example, a head-mounted or hand-held display might present a 3D scene from a perspective that changes depending on the orientation of the display. Augmented reality systems can combine real and virtual objects to allow the two to interact. For example, augmented reality systems can layer digital content on top of real world content appearing in the display. Some augmented reality systems like MICROSOFT HOLOLENS® utilize a headset that allows a user to view the real world through a transparent optical component (such as a lens) while 3D virtual images are projected and reflected toward the user's eye, effectively overlaying the 3D virtual images on top of what the user sees in the real world. A 3D environment can be generated that includes a 3D model of the real world and 3D assets representing virtual objects. As such, the 3D environment can be used to manage the relationship between the 3D assets and the real world, and the 3D assets can be used to project 3D virtual images that appear to interact with the real world.